Intraoral scanning using a pre-existing model

ABSTRACT

An intraoral scanner system includes an intraoral scanner having an imaging device and a sensing face, and a computing device, communicatively coupled to the intraoral scanner. The computing device receives a first intraoral images of a three-dimensional intraoral object of a patient generated by the intraoral scanner corresponding to an intraoral scanning of the three-dimensional intraoral object of the patient. The computing device registers a first intraoral image of the first intraoral images relative to a second intraoral image of the first intraoral images using a model of the three-dimensional intraoral object that existed prior to the intraoral scanning.

This application is a continuation of U.S. patent application Ser. No.16/569,514, filed Sep. 12, 2019, which is a continuation of U.S. patentapplication Ser. No. 14/563,714, filed 8 Dec. 2014, issued as U.S. Pat.No. 10,453,269 on Oct. 22, 2019, the entire contents of all are herebyincorporated by reference herein.

TECHNICAL FIELD

Embodiments of the present invention relate to the field of intraoralscanning and, in particular, to a system and method for using ultrasoundand optical scan data in intraoral scanning.

BACKGROUND

In prosthodontic procedures designed to implant a dental prosthesis inthe oral cavity, the dental site at which the prosthesis is to beimplanted may be measured accurately and studied carefully, so that aprosthesis such as a crown, denture or bridge, for example, can beproperly designed and dimensioned to fit in place. A good fit, forexample, enables mechanical stresses to be properly transmitted betweenthe prosthesis and the jaw and minimizes infection of the gums via theinterface between the prosthesis and the dental site.

Some procedures call for removable prosthetics to be fabricated toreplace one or more missing teeth, such as a partial or full denture, inwhich case the surface contours of the areas where the teeth are missingmay be reproduced accurately so that the resulting prosthetic fits overthe edentulous region with even pressure on the soft tissues.

In some practices, the dental site is prepared by a dental practitioner,and a positive physical model of the dental site is constructed.Alternatively, the dental site may be scanned to providethree-dimensional (3D) data of the dental site. In either case, thevirtual or real model of the dental site may be sent to a dental labthat manufactures the prosthesis based on the model. However, if themodel is deficient or undefined in certain areas, or if the preparationwas not optimally configured for receiving the prosthesis, the design ofthe prosthesis may be less than optimal. For example, if the insertionpath implied by the preparation for a closely-fitting coping wouldresult in the prosthesis colliding with adjacent teeth, the copinggeometry may need to be altered to avoid the collision. Further, if thearea of the preparation containing a finish line lacks definition, itmay not be possible to properly determine the finish line and thus theapical edge of the coping may not be properly designed. Indeed, in somecircumstances, the model is rejected and the dental practitioner thenre-scans the dental site, or reworks the preparation, so that a suitableprosthesis may be produced.

In orthodontic procedures, it can be important to provide a model of oneor both dental arches and/or jaw. Where such orthodontic procedures aredesigned virtually, a virtual model of the oral cavity is alsobeneficial. Such a virtual model may be obtained by scanning the oralcavity directly, or by producing a physical model of the dentition, andthen scanning the model with a suitable scanner.

Thus, in both prosthodontic and orthodontic procedures, obtaining a 3Dmodel of a dental site in the oral cavity may be an initial procedurethat is performed. When the 3D model is a virtual model, the morecomplete and accurate the scans of the dental site are, the higher thequality of the virtual model, and thus the greater the ability to designan optimal prosthesis or orthodontic treatment appliance.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1 illustrates a system for performing intraoral scanning usingultrasound and optical scan data and generating a virtual threedimensional model of a dental site, in accordance with embodiments ofthe present invention.

FIG. 2 illustrates a flow diagram for a method of performing intraoralscanning using ultrasound and optical scan data and generating a virtualthree dimensional model of a dental site, in accordance with embodimentsof the present invention.

FIG. 3 illustrates a functional block diagram of an optical-ultrasounddevice, in accordance with embodiments of the present invention.

FIG. 4 illustrates a flow diagram for a method of creating a virtualmodel using ultrasound and optical scan data, in accordance withembodiments of the present invention.

FIG. 5A illustrates a flow diagram for a method of registering andstitching ultrasound and optical images, in accordance with embodimentsof the present invention.

FIG. 5B illustrates a flow diagram for a method of registering images,in accordance with embodiments of the present invention.

FIG. 6A illustrates a portion of an example dental arch during anintraoral scan session using optical scan data, in accordance withembodiments of the present invention.

FIG. 6B illustrates the example dental arch of FIG. 6A during theintraoral scan session after the generation of further intraoral imagesusing optical scan data, in accordance with embodiments of the presentinvention.

FIG. 7 illustrates an example of a patient's jaw during the intraoralscan session using ultrasound and optical scan data, in accordance withembodiments of the present invention.

FIG. 8 is an example of a virtual model of a three-dimensional object,in accordance with embodiments of the present invention.

FIG. 9 illustrates a block diagram of an example computing device, inaccordance with embodiments of the present invention.

DETAILED DESCRIPTION

Described herein is a method and apparatus for improving the quality ofscans, such as intraoral scans taken of dental sites for patients.During a scan session, a user (e.g., a dental practitioner) of a scannermay generate multiple different images (also referred to as scans) of adental site, model of a dental site, or other object. The images may bediscrete images (e.g., point-and-shoot images) or frames from a video(e.g., a continuous scan). Using only optical images may not capture allthe areas of the dental site and typically may be limited to capturingareas above the gum line. In particular, features below the gum line,such as roots, jaw bone, gap between the roots and jaw bone, and analveolar canal may be missing from optical images. Various x-raytechnologies may be used to capture features below the gum line, such ascone beam computerized tomography (CBCT), but such x-ray technologiesare expensive, large and cumbersome, and submit a patient to radiationexposure. Ultrasound imaging of dental sites may be radiation free,comparatively inexpensive, and provide resolution similar or better thanCBCT. Ultrasound imaging for dental sites may have challenges. Forexample, ultrasound images may be inherently noisy. Additionally, in adental setting a patient's jaw may move during the intraoral scan whichmay further contribute to noisy images. Noisy ultrasound images may makestitching together ultrasound images difficult and potentiallyunrepresentative of features of a dental site. The optical images may beprecise and representative of features of the dental site. The captureof ultrasound images and optical images may be synchronized.Registration data from the optical images may be applied to theultrasound images to address the difficulty of stitching noisyultrasound images.

In one embodiment, an intraoral scan may be taken of a dental site for apatient. The intraoral scan may include optical scans of features abovethe gum line and ultrasound scans of features below the gum line. Aprobe containing both an optical sensor and ultrasound transducer may beused to capture the raw scan data. The optical and ultrasound scans maybe synchronized. For example, an optical and ultrasound scan may besynchronized so that the optical scan may capture the crown of a toothwhile the ultrasound scan may capture the root of the same tooth. Onceboth the optical and ultrasound raw scan data are processed into images,the optical images and ultrasound images may be registered and stitchedtogether. For example, two optical images may be registered to determinerotations and translations for the optical images. Since the opticalimages may have corresponding synchronized ultrasound images from anultrasound transducer having a known fixed offset from an opticalsensor, the rotations and translations of the optical images may beapplied to the corresponding ultrasound images. The optical andultrasound images may be stitched together using the registration dataand an accurate model of the dental site may be created.

Embodiments described herein are discussed with reference to intraoralscanners, intraoral images, intraoral scan sessions, and so forth.However, it should be understood that embodiments also apply to othertypes of scanners than intraoral scanners. Embodiments may apply to anytype of scanner that takes multiple optical and ultrasound images andstitches these images together to form a combined image or virtualmodel. For example, embodiments may apply to desktop model scanners, andso forth. Additionally, it should be understood that the intraoralscanners or other scanners may be used to scan objects other than dentalsites in an oral cavity. Accordingly, embodiments describing intraoralimages should be understood as being generally applicable to any typesof images generated by a scanner that contains both an ultrasoundtransducer and an optical sensor, embodiments describing intraoral scansessions should be understood as being applicable to scan sessions forany type of object, and embodiments describing intraoral scanners shouldbe understood as being generally applicable to many types of scanners.

FIG. 1 illustrates one embodiment of a system 100 for performingintraoral scanning using ultrasound and optical scan data and/orgenerating a virtual three dimensional model of a dental site. In oneembodiment, system 100 carries out one or more operations of belowdescribed in methods 200, 400, and/or 500. System 100 includes acomputing device 105 that may be coupled to a scanner 150 and/or a datastore 110.

Computing device 105 may include a processing device, memory, secondarystorage, one or more input devices (e.g., such as a keyboard, mouse,tablet, and so on), one or more output devices (e.g., a display, aprinter, etc.), and/or other hardware components. Computing device 105may be connected to a data store 110 either directly or via a network.The network may be a local area network (LAN), a public wide areanetwork (WAN) (e.g., the Internet), a private WAN (e.g., an intranet),or a combination thereof. The computing device 105 may be integratedinto the scanner 150 in some embodiments to improve performance andmobility.

Data store 110 may be an internal data store, or an external data storethat is connected to computing device 105 directly or via a network.Examples of network data stores include a storage area network (SAN), anetwork attached storage (NAS), and a storage service provided by acloud computing service provider. Data store 110 may include a filesystem, a database, or other data storage arrangement. Data store 110may include scan data 135, reference data 138, and inertial measurementdata 139. Inertial measurement data 139 may be data from inertialmeasurement module 130.

In some embodiments, a scanner 150 for obtaining three-dimensional (3D)and/or two-dimensional (2D) data of a dental site in a patient's oralcavity is operatively connected to the computing device 105. Scanner 150may include a probe (e.g., a hand held probe) for optically andultrasonically capturing three-dimensional structures (e.g., by confocalfocusing of an array of light beams and by an ultrasound pulse-echoapproach). Scanner 150 may include an optical sensor for opticallycapturing teeth or other visible structures. Scanner 150 mayadditionally include an ultrasound transducer for ultrasonicallycapturing features, such as features below the gum line such as roots,jaw bone, the gap between the root and jaw bone, and the alveolar canal.

The scanner 150 may be used to perform an intraoral scan of a patient'soral cavity. An intraoral scan application 108 running on computingdevice 105 may communicate with the scanner 150 to effectuate theintraoral scan. A result of the intraoral scan may be a sequence ofintraoral images that have been discretely generated (e.g., by pressingon a “generate image” button of the scanner for each image) by, forexample, optical processing module 122 and/or ultrasound processingmodule 124. The images may include optical images and/or ultrasoundimages of the patient's oral cavity. Alternatively, a result of theintraoral scan may be one or more videos of the patient's oral cavity.An operator may start recording the video with the scanner 150 at afirst position in the oral cavity, move the scanner 150 within the oralcavity to a second position while the video is being taken, and thenstop recording the video. In some embodiments, recording may startautomatically as the scanner identifies a tooth, root, or any other areaof interest. The scanner 150 may transmit raw scan data to intraoralscan application 108. The raw scan data may include raw optical scandata and/or raw ultrasound scan data. Raw optical scan data may beprocesses by optical processing module 122 to generate discrete opticalintraoral images or optical intraoral video. Raw ultrasound data may beprocessed by ultrasound processing module 124 to generate discreteultrasound intraoral images or ultrasound intraoral video. Scan data 135may include raw scan data, such as raw optical scan data and rawultrasound scan data, and discrete images and/or intraoral video, suchas discrete optical intraoral image, discrete ultrasound images, opticalintraoral video, and/or ultrasound intraoral video. Optical scan data136 may include any or all of optical scan data such as optical raw scandata, optical intraoral images, and optical intraoral video. Ultrasoundscan data 137 may include any or all ultrasound scan data such asultrasound raw scan data, ultrasound intraoral images, and ultrasoundintraoral video. Scan data 135 may also be referred to collectively asimage data. Images or intraoral images may refer to discrete imagesand/or intraoral video, such as discrete optical intraoral image,discrete ultrasound images, optical intraoral video, and/or ultrasoundintraoral video. Computing device 105 may store the scan data 135 indata store 110. Alternatively, scanner 150 may be connected to anothersystem that stores the image data in data store 110. In such anembodiment, scanner 150 may not be connected to computing device 105.

According to an example, a user (e.g., a practitioner) may subject apatient to intraoral scanning. In doing so, the user may apply scanner150 to one or more patient intraoral locations. The scanning may bedivided into one or more segments. As an example the segments mayinclude a distal buccal region of the patient, a distal lingual regionof the patient, a mesial buccal region of the patient, an mesial lingualregion of the patient, one or more preparation teeth of the patient(e.g., teeth of the patient to which a dental device such as a crown oran orthodontic alignment device will be applied), one or more teethwhich are contacts of preparation teeth (e.g., teeth not themselvessubject to a dental device but which are located next to one or moresuch teeth or which interface with one or more such teeth upon mouthclosure), patient bite (e.g., scanning performed with closure of thepatient's mouth with scan being directed towards an interface area ofthe patient's superior jaw (e.g., upper teeth) and inferior jaw (e.g.,lower teeth), and one or more root and/or jaw features associated withthe aforementioned segments. Via such scanner application, the scanner150 may provide image data (e.g., raw scan data) to computing device105. The raw scan data may be processed to generate images (e.g.,intraoral images) such as, 2D optical intraoral images, 3D opticalintraoral images, 2D ultrasound intraoral images, and 3D ultrasoundintraoral images. Optical raw scan data may be provided from scanner 105to the computing device 105 in order for optical processing module 122to generate optical images in the form of one or more points (e.g., oneor more pixels and/or groups of pixels). For instance, the opticalprocessing module 122 may generate a 2D or 3D optical image as one ormore point clouds. In one embodiment, optical processing module 122 maybe part of scanner 150 and scanner 150 may provide optical image tocomputing device 105. Ultrasound raw scan data may be provided fromscanner 105 to the computing device 105 in order for ultrasoundprocessing module 124 to generate a 2D or 3D ultrasound image. In oneembodiment, ultrasound processing module 124 may be part of scanner 150and scanner 150 may provide ultrasound images to computing device 105.

The manner in which the oral cavity of a patient is to be scanned maydepend on the procedure to be applied thereto. For example, if asuperior or inferior denture is to be created, then a full scan of themandibular or maxillary edentulous arches as well as any associatedfeatures below the gum line (e.g., root and/or jaw features) may beperformed. In contrast, if a bridge is to be created, then just aportion of a total arch and related features below the gum line may bescanned which may include an edentulous region, the neighboring abutmentteeth and roots and the opposing arch and dentition. Thus, the dentalpractitioner may input the identity of a procedure to be performed intointraoral scan application 108. For this purpose, the dentalpractitioner may choose the procedure from a number of preset options ona drop-down menu or the like, from icons or via any other suitablegraphical input interface. Alternatively, the identity of the proceduremay be input in any other suitable way, for example by means of presetcode, notation or any other suitable manner, intraoral scan application108 having been suitably programmed to recognize the choice made by theuser.

By way of non-limiting example, dental procedures may be broadly dividedinto prosthodontic (restorative) and orthodontic procedures, and thenfurther subdivided into specific forms of these procedures.Additionally, dental procedures may include identification and treatmentof gum disease, sleep apnea, and intraoral conditions. The termprosthodontic procedure refers, inter alia, to any procedure involvingthe oral cavity and directed to the design, manufacture or installationof a dental prosthesis at a dental site within the oral cavity, or areal or virtual model thereof, or directed to the design and preparationof the dental site to receive such a prosthesis. A prosthesis mayinclude any restoration such as crowns, veneers, inlays, onlays, andbridges, for example, and any other artificial partial or completedenture. The term orthodontic procedure refers, inter alia, to anyprocedure involving the oral cavity and directed to the design,manufacture or installation of orthodontic elements at a dental sitewithin the oral cavity, or a real or virtual model thereof, or directedto the design and preparation of the dental site to receive suchorthodontic elements. These elements may be appliances including but notlimited to brackets and wires, retainers, clear aligners, or functionalappliances.

A type of scanner to be used may also be input into intraoral scanapplication 108, typically by a dental practitioner choosing one among aplurality of options. If the scanner 150 that is being used is notrecognizable by intraoral scan application 108, it may nevertheless bepossible to input operating parameters of the scanner thereto instead.For example, the optimal spacing between a head of the scanner andscanned surface can be provided, as well as the capture area (and shapethereof) of the dental surface capable of being scanned at thisdistance. Alternatively, other suitable scanning parameters may beprovided. Scanner 150, according to one implementation, may be describedin greater detail with respect to FIG. 3.

Referring back to FIG. 1, intraoral scan application 108 may identifyspatial relationships that are suitable for scanning the dental site sothat complete and accurate image data may be obtained for the procedurein question. Intraoral scan application 108 may establish an optimalmanner for scanning a target area of the dental site.

Intraoral scan application 108 may identify or determine a scanningprotocol by relating the type of scanner, resolution thereof, capturearea at an optimal spacing between the scanner head and the dentalsurface to the target area, etc. For a point-and-shoot scanning mode,the scanning protocol includes a series of scanning stations spatiallyassociated with the dental surfaces of the target area. Preferably,overlapping of the images or scans capable of being obtained at adjacentscanning stations is designed into the scanning protocol to enableaccurate image registration, so that intraoral images can be stitchedtogether to provide a composite 3D virtual model. For a continuousscanning mode (video scan), scanning stations may not be identified.Instead, a practitioner may activate the scanner and proceed to move thescanner within the oral cavity to capture a video of the target areafrom multiple different viewpoints.

In one embodiment, intraoral scan application 108 includes opticalprocessing module 122, ultrasound processing module 124, stitchingmodule 112, synchronization module 120, and inertial measurement module130. Stitching module 112 may include image stitching module 126 andmodel generation module 128. Alternatively, the operations of one ormore of the above-mentioned modules may be combined into a single moduleand/or divided into multiple modules.

Synchronization module 120 may synchronize the optical sensor andultrasound transducer to obtain raw scan data at approximately the sametime. An ultrasound image may correspond to an optical image when thegeneration of the raw scan data for the images is synchronized. Theregistration data from the registration of two optical images may beapplied to corresponding ultrasound images. Registration data mayinclude rotations and translations for the images on which registrationmay be performed. For example, by synchronizing the generation of rawdata the optical image of a tooth may be registered and the registrationdata used to stitch together corresponding ultrasound images.Additionally, the optical images may be stitched together with theultrasound images based on known fixed positional relationship betweenthe optical sensor and the ultrasound transducer. Stitching may be thecombination of two or more images (e.g., optical images, ultrasoundimages, or a combination of both) into a single image using registrationdata. In one embodiment, stitching includes applying the registrationdata from registering the optical images to the corresponding ultrasoundimages. The ultrasound image may be of the tooth's root and associatedpart of the jaw bone, while the optical image may be of a visibleportion of the tooth that is above the gum line (e.g., the crown).Additionally, synchronization module 120 may use any number ofcompensation techniques in order to correct images for any asynchronousdata collection.

Inertial measurement module 130 may receive raw inertial measurementdata from an inertial measurement device in, for example, the probe ofscanner 150. An inertial measurement device may include a combination ofone or more accelerometers and/or one or more gyroscopes, that may helpdetermine the probes velocity, orientation, and/or gravitational forces.Additionally, an inertial measurement device may include a magneticsensor. A magnetic sensor may allow the inertial measurement device todetermine the rotation position of the probe. Raw inertial measurementdata may be processed from inertial measurement module 130 to helpdetermine the orientation of scanner 150 during an intraoral scan. Theorientation of the probe may be used to help determine additionalregistration data. Raw inertial measurement data and processed inertialmeasurement data may be referred to as inertial measurement data and bestored in inertial measurement data 139 of data store 110.

When a scan session is complete (e.g., all images for a dental site havebeen captured), stitching module 112 may generate a virtual 3D model ofthe scanned dental site. Alternatively, stitching module 112 may begingenerating the virtual 3D model as the images are captured (e.g., beforethe scan session is complete). To generate the virtual model, imagestitching module 126 may register the intraoral images generated fromthe intraoral scan session. In one embodiment, performing imagestitching includes capturing 3D data of various points of a surface inmultiple images (views from a camera), and registering the images bycomputing transformations between the images. The images may then beintegrated into a common reference frame (e.g., “stitched” together) byapplying appropriate transformations to points of each registered image.

In one embodiment, image stitching module 126 performing image stitchingmay include registering the optical images together and applying theregistration data from registered optical images to the correspondingultrasound images. Image stitching module 126 may determine that theoptical sensor and the ultrasound transducer of scanner 150 have a fixeddisplacement from one another during an intraoral scan. Image stitchingmodule 126 may register the optical images to determine opticalregistration information (e.g., registration data including rotationsand translations). Additional ultrasound registration data may bedetermined for the ultrasound images by taking account of the fixeddisplacement and/or orientation of the ultrasound transducer from theoptical transducer and modifying the corresponding optical registrationdata to account for the fixed displacement and/or orientation.Alternatively, the image registration data for a pair of optical imagesmay be applied to the corresponding pair of ultrasound images withoutmodification to the image registration data.

Image registration may be performed for each pair of adjacent oroverlapping optical intraoral images (e.g., each successive frame of anintraoral video). Image registration algorithms are carried out toregister two adjacent optical intraoral images, which essentiallyinvolves determination of the transformations which align one image withthe other. Image stitching for ultrasound images may be performed basedon the registration data obtained from registering the optical images.The registration data from registering the optical images may be appliedto the ultrasound images to stitch the ultrasound images together and/orstitch the ultrasound images to the optical images. Each registrationbetween a pair of images may be accurate to within 10-15 microns in oneembodiment. Image registration may involve identifying multiple pointsin each image (e.g., point clouds) of an image pair, surface fitting tothe points of each image, and using local searches around points tomatch points of the two adjacent images. For example, image stitchingmodule 126 may match points of one image with the closest pointsinterpolated on the surface of the other image, and iteratively minimizethe distance between matched points. Image stitching module 126 may alsofind the best match of curvature features at points of one image withcurvature features at points interpolated on the surface of the otherimage, without iteration. Image stitching module 126 may also find thebest match of spin-image point features at points of one image withspin-image point features at points interpolated on the surface of theother image, without iteration. Other techniques that may be used forimage registration include those based on determining point-to-pointcorrespondences using other features and minimization ofpoint-to-surface distances, for example. Other image registrationtechniques may also be used. The optical images and ultrasound imagesmay also be stitched together based on a known offset and/or anglebetween the optical sensor and the ultrasound transducer in the scanner150.

Many image registration algorithms perform the fitting of a surface tothe points in adjacent images, which can be done in numerous ways.Parametric surfaces such as Bezier and B-Spline surfaces are mostcommon, although others may be used. A single surface patch may be fitto all points of an image, or alternatively, separate surface patchesmay be fit to any number of a subset of points of the image. Separatesurface patches may be fit to have common boundaries or they may be fitto overlap. Surfaces or surface patches may be fit to interpolatemultiple points by using a control-point net having the same number ofpoints as a grid of points being fit, or the surface may approximate thepoints by using a control-point net which has fewer number of controlpoints than the grid of points being fit. Various matching techniquesmay also be employed by the image registration algorithms.

In one embodiment, image stitching module 126 may determine a pointmatch between images, which may take the form of a two dimensional (2D)curvature array. A local search for a matching point feature in acorresponding surface patch of an adjacent image is carried out bycomputing features at points sampled in a region surrounding theparametrically similar point. Once corresponding point sets aredetermined between surface patches of the two images, determination ofthe transformation between the two sets of corresponding points in twocoordinate frames can be solved. Essentially, an image registrationalgorithm may compute a transformation between two adjacent images thatwill minimize the distances between points on one surface, and theclosest points to them found in the interpolated region on the otherimage surface used as a reference.

Image stitching module 126 repeats image registration for all adjacentimage pairs of a sequence of intraoral images to obtain a transformationbetween each pair of images, to register each image with the previousone. For each optical image pair, the set of transformations (e.g.,registration data including rotations and translations) that achieve theimage registration are recorded and then applied (e.g., stitched) to thecorresponding pair of ultrasound images. The set of transformations mayor may not be modified based on known offset and/or angle differencesbetween the optical sensor and the ultrasound transducer of the scanner150. Although registering adjacent optical images is discussed, itshould be appreciated that an adjacent image pair may include an opticalimage pair, an ultrasound image pair, and/or and an optical andultrasound image pair. Image stitching module 126 then stitches allimages together by applying the appropriate determined transformationsto each of the optical images and/or ultrasound images. Once the imagesare stitched together the resulting 3D data may be merged to create asingle virtual 3D model. Merging may be the smoothing of the resulting3D data to create a single contiguous virtual 3D model. Merging may beperformed by model generation module 128. As discussed above, in oneembodiment the registration data from the optical images is applied tothe corresponding ultrasound images. In another embodiment, the opticalregistration data is modified to account for the fixed displacementbetween the optical sensor and ultrasound transducer then applied to thecorresponding ultrasound images. Each transformation may includerotations about one to three axes and translations within one to threeplanes.

In another embodiment, image stitching module 126 may register intraoralimages by registering 2D or 3D optical images to a pre-existing model.Registering the optical images may include registering the firstintraoral image using at least a portion of a pre-existing model andregistering a second intraoral image using at least a portion of thepre-existing model. Rotations and translation may be determined based onregistering the optical images to the pre-existing model. Theregistration process may be similar to the process discussed above. Therotations and translations determined from registering the opticalimages may be applied to the ultrasound images in a stitching process ina similar manner as discussed above. The stitched optical and ultrasoundimages may be merged and a virtual model may be created from the merged3D data (e.g., merged image).

The pre-existing model may include a 3D model of the target dental siteand be generated by data stored in reference data 138. Reference data138 may include past data regarding the at-hand patient (e.g., intraoralimages and/or virtual 3D models), some or all of which may be stored indata store 110. The data regarding the at-hand patient may includeX-rays, 2D intraoral images, 3D intraoral images, 2D models, and/orvirtual 3D models corresponding to the patient visit during which thescanning occurs. The data regarding the at-hand patient may additionallyinclude past X-rays, 2D intraoral images, 3D intraoral images, 2Dmodels, and/or virtual 3D models of the patient (e.g., corresponding topast visits of the patient and/or to dental records of the patient). Thereference data 138 may be pooled patient data and may include X-rays, 2Dintraoral images, 3D intraoral images, 2D models, and/or virtual 3Dmodels regarding a multitude of patients. The reference data 138 mayinclude pedagogical patient data. The pedagogical patient data mayinclude X-rays, 2D intraoral images, 3D intraoral images, 2D models,virtual 3D models, and/or medical illustrations (e.g., medicalillustration drawings and/or other images) employed in educationalcontexts. The pedagogical patient data may include volunteer data and/orcadaveric data.

FIG. 2 illustrates a flow diagram for a method of performing intraoralscanning using ultrasound and optical scan data and generating a virtualthree dimensional model of a dental site, in accordance with embodimentsof the present invention. Method 200 may be performed by processinglogic that may comprise hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (such as instructions runon a processing device), firmware, or a combination thereof. In oneimplementation, method 200 may be performed by system 100 of FIG. 1. Inanother implementation, method 200 may be performed or caused to beperformed all or in part by stitching module 112 or intraoral scanapplication 108 of FIG. 1. For simplicity of explanation, method 200 isdepicted and described as a series of acts. However, acts in accordancewith this disclosure can occur in various orders, concurrently, and/orwith other acts not presented or described herein. Furthermore, not allillustrated acts may be required to implement method 200 in accordancewith the disclosed subject matter. In addition, those skilled in the artwill understand and appreciate that method 200 may alternatively berepresented as a series of interrelated states via a state diagram orinterrelated events.

Method 200 begins at block 201 where processing logic implementing themethod may generate and obtain raw optical scan data from an opticalsensor attached to an optical probe. The optical probe may be part ofscanner 150. The raw optical scan data may be 3D optical scan data or 2Doptical scan data. The optical probe may be synchronized with theultrasound probe in block 204 so that the ultrasound probe 204 generatesultrasound scans synchronously to the generation of the optical scans(e.g., so that the separate scans may occur simultaneously). The opticalprobe and ultrasound probe may be in the same probe or in differentprobes. The optical probe (i.e., optical sensor) and the ultrasoundprobe (i.e., ultrasound transducer) may be disposed in the probe with afixed displacement between the two so that they move together during anintraoral scan.

Method 200 continues to block 202 where processing logic implementingthe method processes the raw optical scan data to generate 2D or 3Doptical images. Optical processing to generate optical images may bediscussed in more detail with respect to FIGS. 1 and 3.

At the time of generating the optical scans (block 201) and ultrasoundscans (block 204), an inertial measurement unit 203 may collect rawinertial measurement data of the optical and ultrasound probe. Theinertial measurement unit 203 may include an inertial measurementsensor, an inertial measurement module, and inertial measurement data.The inertial measurement unit 203 may be located in the ultrasoundprobe, optical probe, or scanner 150. The raw inertial measurement datamay be processed by inertial measurement unit to determine theorientation of scanner 150, and more specifically the orientation of theoptical probe and ultrasound probe. The orientation information may beused as an additional data point to aid in image registration, and theoptical images and ultrasound images may be more accurately registeredtogether. The collection of raw inertial measurement data may besynchronized with the intraoral scanning, such as with the optical scanand ultrasound scan.

As mentioned, at block 204 processing logic generates and obtains rawultrasound scan data from an ultrasound transducer attached to anultrasound probe. The raw optical scan data may be 3D ultrasound scandata or 2D ultrasound scan data. The ultrasound probe may besynchronized with the optical probe in block 201, as discussed above.Blocks 201 and 204 may be performed synchronously, so that the opticaland ultrasound scans may occur approximately simultaneously (e.g.,accounting for differing measuring speeds of the optical probe versesthe ultrasound probe).

Method 200 continues to block 205 where processing logic implementingthe method processes the raw ultrasound scan data to generate 2D or 3Dultrasound images. Ultrasound processing to generate ultrasound imagesmay be discussed in more detail with respect to FIGS. 1 and 3.

Method 200 continues to block 206 where processing logic implementingthe method receives optical images and ultrasound images. The opticalimages may be used to produce registration information (i.e.,registration data), such as rotations and translations. In oneembodiment, the ultrasound images may be used to produce additionalregistration information. Further information, such as scannerorientation, may be received from the inertial measurement unit in block203. In another embodiment, optical images may be registered to apre-existing model, as discussed in regards to FIG. 1, to produceregistration data. For example, 2D optical images may be taken, and maybe registered against a virtual 3D model of a patient's dental arch togenerate the registration data.

Method 200 continues to block 207 where processing logic implementingthe method may stich and merge the optical and/or ultrasound images asdiscussed above with reference to FIG. 1. Using the registration datareceived at block 206, the optical and/or ultrasound images may bestitched together to form an image that includes features above andbelow the gum line at block 207. The optical scan data (e.g., opticalimages) as discussed in reference to block 202 may be sent to block 207from block 202 and/or block 206. Registration data based on the opticalscan data may be used to stitch together the optical images at block207. Additionally, the registration data based on the optical scan datamay be used to stitch together the ultrasound images. Once separateimage are stitched together, the resulting 3D data may be merged (e.g.,smoothed) to create a 3D virtual model.

In an alternative embodiment, the ultrasound images may be registeredand stitched together without the use of registration information fromthe optical images, as illustrated in FIG. 2 by the ultrasound imagesbeing sent directly to block 207 from block 205. Registration data maybe generated for the ultrasound images themselves. Additionalregistration data for the ultrasound images may be generated by theinertial measurement unit, as described in block 203. The ultrasoundimages may be stitched together using the registration data.Additionally, the optical images may be stitched to the ultrasoundimages using the registration data generated from the ultrasound imagesand/or inertial measurement unit.

In another embodiment, the optical sensor as described in reference toblock 201 and the ultrasound transducer as described in reference toblock 204 may be in different probes and/or not synchronized. In such acase, a fiducial may be used in conjunction with an optical scan and anultrasound scan. A fiducial may be an object placed in the field of viewof an imaging system, such as an optical probe and/or an ultrasoundprobe, which appears in the image produced. The fiducial may be used asa point of reference or a measure. For example, a fiducial may be apowder that includes particles that reflect light and may be detected byan optical probe and an ultrasound probe. A fiducial may be an adhesiveobject that may be place at the dental site and be detected by anoptical probe and an ultrasound probe.

The fiducial may be used to aid in image registration. For example, thefiducial may provide a geometrical and/or optical/ultrasound referencepoint for image registration of optical images, ultrasound images,and/or optical images together with ultrasound images. Duringregistration between two intraoral images, each image may contain atleast a portion of the fiducial (e.g., an optical image andcorresponding ultrasound image each contain at least a portion of thefiducial), so the registration of the two intraoral images may havesufficient accuracy. By having known properties, a fiducial mayfacilitate (e.g., increase accuracy) of the registration data.Additionally, the portion of the image which includes the fiducial, orparts thereof, may be subtracted out from the scanned object, therebyleaving behind the original object's shape without the presence of thefiducial. Once the registration data is generated for the at least twoimages, the images may be stitched together and merged as described inreference to block 207.

Method 200 continues to block 208 where processing logic implementingthe method may create a virtual model, such as a 3D virtual model, ofthe dental site (e.g., target area). In the case of a virtual model of adental site, the virtual model may include all or some of the teeth,roots, gap between the roots and the bone, features of the jaw bone,and/or the alveolar canal. The 3D virtual model may be manipulated sothat layers may be added or subtracted. For example, a practitioner mayonly want to look at the teeth and roots and may remove the jaw layerfrom the 3D virtual model, remove the gums from the 3D virtual model,etc. Layers may be displayed or removed from display based on one ormore filter options.

FIG. 3 illustrates a functional block diagram of an optical-ultrasounddevice, in accordance with some embodiments of the present invention. Anoptical-ultrasound device may be a scanner, for example scanner 150.Scanner 150 may be an intraoral scanner. Scanner 150 may include anoptical imaging device 320 and ultrasound imaging device 395. Forpurposes of illustration, 3D axis 350 is included in FIG. 3 but may notbe part of scanner 150.

Optical imaging device 320 includes a semiconductor laser 328 that emitsa laser light (represented by the arrow 330). The light passes through apolarizer 332 which gives rise to a certain polarization of the lightpassing through polarizer 332. The light then enters into an opticexpander 334 that improves the numerical aperture of the light beam 330.The light then passes through a module 338 (e.g., a grating or a microlens array) that splits the parent beam 330 into multiple incident lightbeams 336, represented in FIG. 3 by a single line for ease ofillustration.

The optical imaging device 320 further includes a partially transparentmirror 340 having a small central aperture. The mirror 340 allowstransfer of light from the laser source through the downstream optics,but reflects light travelling in the opposite direction. In otherembodiments, rather than a partially transparent mirror, other opticalcomponents with a similar function may also be used, e.g. a beamsplitter. The aperture in the mirror 340 improves the measurementaccuracy of the apparatus. As a result of this mirror 340, the lightbeams will yield a light annulus on the illuminated area of the imagedobject as long as the area is not in focus and the annulus will turninto a completely illuminated spot once in focus.

Optical imaging device 320 further includes confocal optics 342operating in a telecentric mode, relay optics 344, and an endoscope 346.In one embodiment, telecentric confocal optics avoid distance-introducedmagnification changes and maintains the same magnification of the imageover a wide range of distances in the Z-direction (the Z-direction beingthe direction of beam propagation, also referred to as the Z-axis orlens axis). The relay optics 344 allow maintenance of a certainnumerical aperture of the beam's propagation.

The endoscope 346 typically includes a rigid, light-transmitting medium.The rigid, light-transmitting medium may be a hollow object definingwithin it a light transmission path or an object made of a lighttransmitting material (e.g., a glass body or tube). At its end, theendoscope typically includes a mirror of the kind ensuring a totalinternal reflection. The mirror may direct incident light beams towardsa teeth segment 326 that is being scanned. The endoscope 346 thus emitsmultiple incident light beams 348 impinging on to the surface of theteeth of dental site 326.

It should be noted that scanner 150 may include a probe with a sensingface. The probe may be an end portion of scanner 150 such as a portionof optical imaging device 320 and ultrasound imaging device 395 thatincludes the endoscope 346 and/or ultrasound transducer 392. The opticalsensor may include all or part of imaging device 320. The optical sensor320 may alternatively be an end portion of the optical imaging device320 that includes the endoscope 346. The probe portion of scanner 150may be the portion of scanner 150 that extends to the dental site 326.The sensing face of scanner may be a side of the scanner 150 that is incontact with the dental site 326. For example the face of scanner 150where endoscope 346 and ultrasound transducer 392 are directed towardsdental site 326 may be the sensing face. A fixed displacement 396 isillustrated between ultrasound transducer 392 and the optical sensor(e.g., endoscope 346). In one embodiment, scanner 150 is configured suchthat the ultrasound transducer 392 is to be pressed against a patient'sgum while the endoscope is positioned proximate to a crown of apatient's tooth. Thus, the optical imaging device 320 may generate anoptical image of the tooth above the gum line, and the ultrasoundimaging device may synchronously generate an ultrasound image of thatsame tooth below the gum line.

The incident light beams 348 form an array of light beams arranged in anX-Y plane propagating along the Z-axis. If the surface on which theincident light beams hit is an uneven surface, illuminated spots 352 aredisplaced from one another along the Z-axis, at different (Xi, Yi)locations. Thus, while a spot at one location may be in focus of theoptical element 342, spots at other locations may be out-of-focus.Therefore, the light intensity of the returned light beams (see below)of the focused spots will be at its peak, while the light intensity atother spots will be off peak. Thus, for each illuminated spot, multiplemeasurements of light intensity are made at different positions alongthe Z-axis. For each of such (Xi, Yi) location, typically the derivativeof the intensity over distance (Z) will be made, the Z, yielding maximumderivative, Zo, will be the in-focus distance. As pointed out above,where, as a result of use of the partially transparent mirror 340, theincident light forms a light disk on the surface when out of focus and acomplete light spot only when in focus, the distance derivative will belarger when approaching in-focus position thus increasing accuracy ofthe measurement.

The light scattered from each of the light spots includes a beamtravelling initially in the Z-axis along the opposite direction of theoptical path traveled by the incident light beams. Each returned lightbeam 354 corresponds to one of the incident light beams 336. Given theunsymmetrical properties of the mirror 340, the returned light beams arereflected in the direction of the detection optics 360. The detectionoptics 360 may include a polarizer 362 that has a plane of preferredpolarization oriented normal to the plane polarization of polarizer 332.The returned polarized light beam 354 passes through an imaging optic364, typically one or more lenses, and then through a matrix 366including an array of pinholes. A CCD (charge-coupled device) camera 368has a matrix of sensing elements each representing a pixel of the imageand each one corresponding to one pinhole in the array 366.

The CCD camera 368 is connected to the communication module 391.Communication module 391 may coordinate the communication between andreceived from optical imaging device 320, ultrasound imaging device 395,inertial measurement device 390, and computing device 105. For example,communication module may synchronize the collection of data between theoptical imaging device 320, the ultrasound imaging device 395, andinertial measurement device 390. Communication module 391 may beconnected to image-capturing module 380 of computing device 105.Computing device 105 may be a processing device. Thus, each lightintensity measurement in each of the sensing elements of the CCD camera368 may be received and analyzed by computing device 105.

The optical imaging device 320 further includes a control module 370connected to a controlling operation of both the semiconductor laser 328and an actuator 372. The actuator 372 is linked to the telecentricconfocal optics 342 to change the relative location of the focal planeof the confocal optics 342 along the Z-axis. In a single sequence ofoperation, the control module 370 induces the actuator 372 to displacethe confocal optics 342 to change the focal plane location and then,after receipt of a feedback that the location has changed, the controlmodule 370 will induce the laser 328 to generate a light pulse. At thesame time, the control module 370 will synchronize the communicationmodule 391 or image capturing module 380 to collect data representativeof the light intensity from each of the sensing elements of the CCDcamera 368. Then, in subsequent sequences the focal plane will change inthe same manner and the data capturing will continue over a wide focalrange.

The image capturing device 380 is connected to intraoral scanapplication 108 which then determines the relative intensity in eachpixel over the entire range of focal planes of optics 342, 344. Asexplained above, once a certain light spot is in focus, the measuredintensity will be maximal. Thus, by determining the Z, corresponding tothe maximal light intensity or by determining the maximum displacementderivative of the light intensity, for each pixel, the relative positionof each light spot along the Z-axis can be determined. Thus, datarepresentative of the three-dimensional pattern of a surface in theteeth segment can be obtained. This three-dimensional representation(e.g., image) may be displayed on a display 384 and manipulated forviewing, e.g. viewing from different angles, zooming-in or out, by auser control module 385 (e.g., a computer keyboard, touchpad, mouse,etc.). In addition, the data representative of the surface topology maybe transmitted through an appropriate data port, e.g. a modem, throughany communication network (e.g., a local area network (LAN), wide areanetwork (WAN), public network such as the Internet, etc.) to arecipient.

Scanner 150 may also include an inertial measurement device 390.Inertial measurement device 390 may include any combination of one ormore accelerometers and/or one or more gyroscopes. Inertial measurementdevice 390 may include one or more magnetic sensor. The magnetic sensormay allow inertial measurement device 390 to detect the rotationposition of scanner 150. The inertial measurement device 390 may be partof the probe of scanner 150. Inertial measurement device 390 may detecta rate of acceleration using one or more accelerometers and may detectchanges in rotational attributes such as pitch, roll, and yaw using oneor more gyroscopes. Raw inertial measurement data, such asaccelerometer, gyroscope, and/or magnetic sensor data, may be sent tocommunication module 391. Communication module 391 may send the rawinertial measurement data to intraoral scan application 108. The rawinertial measurement data may be processed to aid in determine of theposition of the probe of scanner 150, which may be used as additionalinput for generation of registration data. For example, intraoral scanapplication 108 may continually calculate the probes current positionwhile performing intraoral scanning. For each of the six degrees offreedom, (x, y, z, θx, θy, and θz), intraoral scan application 108 mayintegrate over time the sensed acceleration, together with an estimateof gravity, calculate the current velocity. The current velocity may beintegrated to calculate the current position.

Ultrasound imaging device 395 may include an ultrasound transducer 392and ultrasound electronics 393. Ultrasound transducer 392 may transmithigh-frequency (e.g., 1 to 20 MHz) sound pulses (e.g., sound waves orultrasound beam) into the dental site 326. A frequency for the soundwave may be selected to provide optimal image resolution. High frequencywaves may be more attenuated than low frequency waves for a givendistance, but may have better resolution. The sound waves travel intothe dental site 326 and impact materials with different acousticimpedances along the path of transmission. Some of the sound waves maybe reflected back to ultrasound transducer 392, while some reach anothermaterial and then get reflected. The reflected waves (i.e., echoes) aredetected by ultrasound transducer 392 and sent as raw ultrasound scandata to communication module 391 and eventually to intraoral scanapplication 108. Intraoral scan application 108 calculates the distancefrom the ultrasound transducer 392 to the dental site 352 using thespeed of sound in a medium and the time of each echo's return. A largenumber, for example millions, of pulses and echoes may be sent andreceived each second. The ultrasound transducer 392 may be moved alongthe gum line and angled to obtain various views of the dental site. Forexample, in dental site 326 ultrasound transducer may be partially abovethe gum line but predominately located below the gum line to detectfeatures hidden from the eye (and from optical imaging device 320), suchas roots and jaw bone. Raw ultrasound scan data may be processed byintraoral scan application 108 to generate ultrasound images of thedental site 326.

An ultrasound transducer 392 may include one or more quartz crystals orpiezoelectric crystals. Electric current may be applied to the crystalsthat may cause the crystals to change shape. The shape changes producesound waves that travel outward. Conversely, when sound or pressurewaves hit the crystals, the crystals may emit electrical currents.Additionally, ultrasound transducer 392 may be an array transducerincluding an array of individual ultrasound transducers. The arraytransducer may scan the target site 326 multiple times by using focusedbeams oriented along converging directions. The echoes from the targetsite may be averaged together into a single image.

Ultrasound electronics 393 may contain electronics to store, send, andreceive electrical signals to ultrasound transducer 392. For instance,ultrasound electronics may include a microprocessor, memory, andamplifiers. The ultrasound electronics 393 may send electrical signalsto ultrasound transducer 392 to emit sound waves, and may also receiveelectrical signals created from returning echoes. Additionally, theultrasound electronics 393 may change the frequency and duration of theultrasound signals.

FIG. 4 illustrates a flow diagram for a method of creating a virtualmodel using ultrasound and optical scan data, in accordance withembodiments of the present invention. Method 400 may be performed byprocessing logic that may comprise hardware (e.g., circuitry, dedicatedlogic, programmable logic, microcode, etc.), software (such asinstructions run on a processing device), firmware, or a combinationthereof. In one implementation, method 400 may be performed by system100 of FIG. 1. In another implementation, method 400 may be performed orcaused to be performed all or in part by stitching module 112 orintraoral scan application 108 of FIG. 1. For simplicity of explanation,method 400 is depicted and described as a series of acts. However, actsin accordance with this disclosure can occur in various orders,concurrently, and/or with other acts not presented or described herein.Furthermore, not all illustrated acts may be required to implementmethod 400 in accordance with the disclosed subject matter. In addition,those skilled in the art will understand and appreciate that method 400may alternatively be represented as a series of interrelated states viaa state diagram or interrelated events.

Method 400 begins at block 402 where processing logic implementing themethod receives optical scan data including a first set of imagesrepresenting a first portion of a three-dimensional object. For example,processing logic may receive raw optical scan data from scanner 150 of adental site. The raw optical scan data may be processed to generate aset of optical images. Processing logic may receive a first set ofoptical images of a dental site. The three-dimensional object may be aportion of a jaw or dental arch and include crowns of one or more teethin the jaw or dental arch. The three-dimensional object may include acoronal portion of a tooth. Coronal may mean the crown of a tooth,towards the crown of a tooth and/or related to the crown of the tooth.The optical images may be 2D images or 3D images. Block 402 may befurther described with respect to FIGS. 1, 2, and 3.

Method 400 continues at block 404 where processing logic implementingthe method receives ultrasound scan data including a second set ofimages representing a second portion of a three-dimensional object. Forexample, processing logic may receive raw ultrasound scan data fromscanner 150 of a dental site. The raw ultrasound scan data may beprocessed to generate a set of ultrasound images. Processing logic mayreceive the ultrasound images of a dental site. The three-dimensionalobject may be a portion of a jaw or dental arch and include roots of oneor more teeth in the jaw or dental arch. The three-dimensional objectmay include an apical portion of a tooth. Apical may mean tooth roots,towards the root tips of a tooth and/or related to the roots. Thegeneration of the optical and ultrasound scan data may be approximatelysynchronized. The optical sensor and the ultrasound transducer may havea fixed displacement. Block 404 may be further described with respect toFIGS. 1, 2, and 3.

Method 400 continues at block 406 where processing logic performs imagestitching between the second set of three-dimensional images using theoptical scan data. In particular, the first set of images in the opticalscan data may be registered together and/or registered to a virtual 3Dmodel. This registration may produce registration data. The registrationdata from the optical images may be applied to corresponding ultrasoundimages in a stitching process. For example, each ultrasound image maycorrespond to an optical image taken at the same time as the ultrasoundimage. The registration data from the optical images may be used tostitch the ultrasound images to one another and/or stitch the opticalimages to the ultrasound images. Block 406 may be further described withrespect to FIGS. 1, 2, 3, and 5.

In one embodiment, the first set of images may be a first set oftwo-dimensional images. Each three-dimensional image of the second setof three-dimensional images may be correlated to a two-dimensional imageof the first set of two-dimensional images. For example, the opticalimages may be 2D optical images. Image registration may includeregistering the first set of 2D optical images to a pre-existing model.The registering may include determining registration data, such asrotations and translations, for one or more of the optical images thatcause features in those optical images to align to similar features inthe pre-existing model. For one of the more 3D images of the 3Dultrasound images, the registration data of a corresponding image fromthe 2D optical images may be applied in a stitching process. Forexample, the 2D optical images may be registered to a pre-existingmodel, such as a model generated from a patient's records, to generateregistration data. In the stitching process, the registration data fromthe 2D optical images may be applied to corresponding 3D ultrasoundimages. Once, the 2D optical images and corresponding 3D ultrasoundimages have been stitched together, the resulting 3D data from thestitched images may be merged. Merging the images may smooth thetransitions between the stitched images.

Method 400 continues at block 408 where processing logic creates avirtual model of the three-dimensional object based on the stitchedsecond set of three-dimensional images. For example, the virtual modelmay be created of the dental site, such as crowns and roots of one ormore teeth. The virtual model may be a 3D virtual model. The ultrasoundimages may be used to create the features below the gum line of thevirtual model. For example, the ultrasound images may be used to createthe roots, jaw bone, gap between the roots and jawbone. An example 3Dvirtual model 800 generated in accordance with method 400 is shown atFIG. 8. Block 408 may be further described with respect to FIGS. 1, 2,and 3.

FIG. 5A illustrates a flow diagram for a method of registering andstitching ultrasound and optical images, in accordance with embodimentsof the present invention. Method 500 and/or 550 may be performed byprocessing logic that may comprise hardware (e.g., circuitry, dedicatedlogic, programmable logic, microcode, etc.), software (such asinstructions run on a processing device), firmware, or a combinationthereof. In one implementation, method 500 and/or 550 may be performedby system 100 of FIG. 1. In another implementation, method 500 and/or550 may be performed or caused to be performed all or in part bystitching module 112 or intraoral scan application 108 of FIG. 1. Forsimplicity of explanation, method 500 and/or 550 is depicted anddescribed as a series of acts. However, acts in accordance with thisdisclosure can occur in various orders, concurrently, and/or with otheracts not presented or described herein. Furthermore, not all illustratedacts may be required to implement method 500 and/or 550 in accordancewith the disclosed subject matter. In addition, those skilled in the artwill understand and appreciate that method 500 and/or 550 mayalternatively be represented as a series of interrelated states via astate diagram or interrelated events.

Method 500 begins at block 502 where processing logic implementing themethod registers the first set of images to each other. The registeringmay include determining registration data, such as rotations andtranslations, to perform on one or more of the first set of opticalimages. For example, processing logic may register optical imagestogether to generate registration data. Processing logic may useinertial measurement data from an inertial measurement device to improvean accuracy of the registration data and/or to improve a processing timeto generate the registration data. Block 502 may be further describedwith respect to FIGS. 1, 2, 3, and 4.

Method 500 continues at block 504 where processing logic applies therotations and translations of a corresponding image from the first setof images for one or more three-dimensional image of the second set ofthree-dimensional images. For example, processing logic may use theregistration data from the optical images and apply the registrationdata to one or more corresponding ultrasound images. Block 504 may befurther described with respect to FIGS. 1, 2, 3, and 4.

Method 500 continues at block 506 where processing logic stitches thesecond set of three-dimensional images with the first set of images. Forexample, processing logic may use the registration data for the opticalimages and the ultrasound images and stitch the optical and ultrasoundimages. Block 506 may be further described with respect to FIGS. 1, 2,3, and 4.

FIG. 5B illustrates a flow diagram for a method of registering images,in accordance with embodiments of the present invention.

Method 550 begins at block 552, processing logic receives a firstplurality of intraoral images of a first portion of a three-dimensionalintraoral object, the first plurality of intraoral images correspondingto an intraoral scan of the three-dimensional intraoral object during acurrent patient visit.

At block 554, processing logic Identifies a pre-existing model thatcorresponds to the three-dimensional intraoral object, the pre-existingmodel based on intraoral data of the three-dimensional object capturedduring a previous patient visit.

At block 556, processing logic registers a first intraoral image of thefirst plurality of intraoral images to a first portion of thepre-existing model.

At block 558, processing logic registers a second intraoral image of thefirst plurality of intraoral images to a second portion of thepre-existing model.

FIG. 6A illustrates a portion of an example dental arch during anintraoral scan session using optical scan data, in accordance withembodiments of the present invention. Dental site 600 may be a scannedportion of a dental arch scanned during an intraoral scan session, inparticular an optical portion of an intraoral scan. The dental site 600includes gums 604 and multiple teeth 610, 620. Multiple opticalintraoral images 625, 630, 635, 640, 645 have been taken of dental site600 of a patient. Each of the optical intraoral images 625-645 may havebeen generated by an intraoral scanner having a particular distance fromthe dental surface being imaged. At the particular distance, the opticalintraoral images 625-645 have a particular scan area and scan depth. Theshape and size of the scan area rill generally depend on the scanner,and is herein represented by a rectangle, Each image may have its ownreference coordinate system and origin. Each intraoral image may begenerated by a scanner, such as scanner 150, at a particular position(scanning station). The location and orientation of scanning stationsmay be selected such that together the optical intraoral images (andultrasound intraoral images) adequately cover an entire target zone.Preferably, scanning stations are selected such that there is overlapbetween the intraoral images 625-645 as shown. Typically, the selectedscanning stations will differ when different scanners are used for thesame target area, depending on the capture characteristics of thescanner used. Thus, a scanner capable of scanning a larger dental areawith each scan (e.g., having a larger field of view) will use fewerscanning stations than a scanner that is only capable of capturing 3Ddata of a relatively smaller dental surface. Similarly, the number anddisposition of scanning stations for a scanner having a rectangularscanning grid (and thus providing projected scanning areas in the formof corresponding rectangles) will typically be different from those fora scanner having a circular or triangular scanning grid (which wouldprovide projected scanning areas in the form of corresponding circles ortriangles, respectively).

FIG. 6B illustrates the example dental arch of FIG. 6A during theintraoral scan session after the generation of further intraoral imagesusing optical scan data, in accordance with embodiments of the presentinvention. Dental site 602 illustrates a scanned portion of a dentalarch, which may be an update of dental site 600. Additional opticalintraoral images 658, 659 have been taken to provide additional opticalimage data, for example for areas that were not clearly captured in FIG.6A. Additional optical intraoral images 660, 662, 664, 666 have alsobeen generated. These additional optical intraoral images 660-666 revealadditional features of dental site 602 such as teeth 450, 452, 454, 456.A practitioner may generate still further intraoral images as desiredand to provide data for a larger target area, such as a full dentalarch. While only the optical images are shown, it should be understoodthat each illustrated optical intraoral image is associated with acorresponding ultrasonic intraoral image that has its own scan area andscan depth.

FIG. 7 illustrates an example of a patient's jaw during the intraoralscan session using ultrasound and optical scan data, in accordance withembodiments of the present invention. Dental site 700 illustrates ascanned portion of a patients jaw during an intraoral scan session.Dental site 700 may be the same scanned site as dental site 600 and 602but also includes images of features below the gum line obtained from anultrasound scan that is synchronized to the optical scan. The intraoralscan session may include both optical and ultrasound images. The dentalsite 700 includes multiple teeth 610, 620, 650, 652, and 654. The dentalsite 700 also includes features below the gum line such as portions ofthe jaw bone, such as jaw bone 712, roots of teeth, such as root 704,and the alveolar canal 702. As described with respect to FIGS. 6A and6B, multiple optical intraoral images 625, 635, 664 have been taken ofdental site 700 of a patient. Additionally, multiple ultrasoundintraoral images 706, 708, 710 have been taken of dental site 700 of thepatient. The generation of the optical scan data and ultrasound scandata may be synchronized so that optical images may correspond withultrasound images. For example, optical image 625 may have beensynchronized with ultrasound image 706, optical image 635 may have beensynchronized with ultrasound image 708, and optical image 664 may havebeen synchronized with ultrasound image 710. The corresponding opticaland ultrasound images may have an overlapping portion which may beillustrated by the overlapping portion of the image frames.Alternatively, there may be no overlap between the corresponding opticaland ultrasound images. Each of the ultrasound intraoral images 706-710may have been generated by an intraoral scanner having a particulardistance from the dental surface being imaged. In one embodiment, aprobe including the ultrasound transducer is pressed against a patient'sgum during imaging. Alternatively, a medium (e.g., a soft medium thatmay have a similar density to the gum) may be interposed between theprobe of the ultrasound transducer and the gum. At the particulardistance, the ultrasound intraoral images 625-645 have a particular scanarea and scan depth. Each of the ultrasound intraoral images 706-710 mayhave been generated by an intraoral scan using a sound wave having aparticular frequency. At the particular frequency, the ultrasoundintraoral images 625-645 have a particular scan area and scan depth. Theshape and size of the scan area will generally depend on the scanner,and is herein represented by a rectangle. Each image may have its ownreference coordinate system and origin. Each intraoral image may begenerated by a scanner, such as scanner 150, at a particular position(scanning station), The location and orientation of scanning stationsmay be selected such that together the optical intraoral images andultrasound intraoral images adequately cover an entire target zone,Preferably, scanning stations are selected such that there is overlapbetween the ultrasound intraoral images 706-710 as shown. Typically, theselected scanning stations will differ when different scanners are usedfor the same target area, depending on the capture characteristics ofthe scanner used.

FIG. 8 is an example of a virtual model of a three-dimensional object,generated in accordance with embodiments of the present invention.Virtual model 800 may be 3D virtual module of the dental site 700depicted in FIG. 7 after intraoral scans have been taken of the entirejaw. Virtual model 800 includes the both the crowns 805 and roots 810 ofteeth as well as partial gums 815. Virtual model 800 may be manipulatedto view dental site 700 at different angles. Virtual model 800 may alsobe manipulated to add or subtract different layers. In virtual model800, the jaw bone has been removed. Additionally, the gums may beremoved. Virtual model 800 may be manipulated to add the jaw bone or anyother features of the dental site 700. By including ultrasound images increating the virtual model, features below the gum line may beaccurately replicated by virtual model 800.

FIG. 9 illustrates a block diagram of an example computing device, inaccordance with embodiments of the present invention. In alternativeimplementations, the machine may be connected (e.g., networked) to othermachines in a LAN, an intranet, an extranet, or the Internet. Themachine may operate in the capacity of a server or a client device in aclient-server network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine may be apersonal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, aserver, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein.

The computer system 900 includes a processing device 902, a main memory904 (e.g., read-only memory (ROM), flash memory, dynamic random accessmemory (DRAM) (such as synchronous DRAM (SDRAM) or DRAM (RDRAM), etc.),a static memory 906 (e.g., flash memory, static random access memory(SRAM), etc.), and a data storage device 918, which communicate witheach other via a bus 930.

Processing device 902 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device may be complex instruction setcomputing (CISC) microprocessor, reduced instruction set computer (RISC)microprocessor, very long instruction word (VLIW) microprocessor, orprocessor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processing device 902may also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The processing device 902 may be configured to execute theprocessing logic 926 for performing the operations and steps discussedherein.

The computer system 900 may further include a network interface device908 communicably coupled to a network 920. The computer system 900 alsomay include a video display unit 910 (e.g., a liquid crystal display(LCD) or a cathode ray tube (CRT)), an alphanumeric input device 912(e.g., a keyboard), a cursor control device 914 (e.g., a mouse), and asignal generation device 916 (e.g., a speaker).

The data storage device 918 may include a machine-accessible storagemedium 924 on which may be stored software 926 embodying any one or moreof the methodologies of functions described herein. The software 926 mayalso reside, completely or at least partially, within the main memory904 as instructions 926 and/or within the processing device 902 asprocessing logic 926 during execution thereof by the computer system900; the main memory 904 and the processing device 902 also constitutingmachine-accessible storage media.

The machine-readable storage medium 924 may also be used to storeinstructions 926 to implement the registration module 122 and/orintraoral scan application 108 to implement any one or more of themethodologies of functions described herein in a computer system, suchas the system described with respect to FIG. 1, and/or a softwarelibrary containing methods that call the above applications.

While the machine-accessible storage medium 924 is shown in an exampleimplementation to be a single medium, the term “machine-accessiblestorage medium” should be taken to include a single medium or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store the one or more sets of instructions. Theterm “machine-accessible storage medium” shall also be taken to includeany medium that may be capable of storing, encoding or carrying a set ofinstruction for execution by the machine and that cause the machine toperform any one or more of the methodologies of the disclosure. The term“machine-accessible storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, and optical andmagnetic media.

In the foregoing description, numerous details are set forth. It may beapparent, however, that the disclosure may be practiced without thesespecific details. In some instances, well-known structures and devicesare shown in block diagram form, rather than in detail, in order toavoid obscuring the disclosure.

Some portions of the detailed descriptions which follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “receiving”, “performing”,“creating”, “registering”, “applying”, “allocating”, “merging”, “using”,or the like, refer to the action and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (electronic) quantities within the computersystem's registers and memories into other data similarly represented asphysical quantities within the computer system memories or registers orother such information storage, transmission or display devices.

The disclosure also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a machinereadable storage medium, such as, but not limited to, any type of diskincluding floppy disks, optical disks, CD-ROMs, and magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, each coupled to a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems may appear as set forth in thedescription below. In addition, the disclosure is not described withreference to any particular programming language. It may be appreciatedthat a variety of programming languages may be used to implement theteachings of the disclosure as described herein.

The disclosure may be provided as a computer program product, orsoftware, that may include a machine-readable medium having storedthereon instructions, which may be used to program a computer system (orother electronic devices) to perform a process according to thedisclosure. A machine-readable medium includes any mechanism for storingor transmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-readable (e.g., computer-readable)medium includes a machine (e.g., a computer) readable storage medium(e.g., read only memory (“ROM”), random access memory (“RAM”), magneticdisk storage media, optical storage media, flash memory devices, etc.),etc.

Whereas many alterations and modifications of the disclosure may nodoubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that anyparticular example shown and described by way of illustration is in noway intended to be considered limiting. Therefore, references to detailsof various examples are not intended to limit the scope of the claims,which in themselves recite only those features regarded as thedisclosure.

What is claimed is:
 1. An intraoral scanner system comprising: anintraoral scanner comprising an imaging device and a sensing face; and acomputing device, communicatively coupled to the intraoral scanner, to:receive a first plurality of intraoral images of a three-dimensionalintraoral object of a patient generated by the intraoral scannercorresponding to an intraoral scanning of the three-dimensionalintraoral object of the patient; and register a first intraoral image ofthe first plurality of intraoral images relative to a second intraoralimage of the first plurality of intraoral images using a model of thethree-dimensional intraoral object that existed prior to the intraoralscanning.
 2. The intraoral scanner system of claim 1, wherein the modelis based on intraoral data representative of the three-dimensionalintraoral object of the patient.
 3. The intraoral scanner system ofclaim 1, the computing device further to: identify the model of thethree-dimensional intraoral object stored at a data store.
 4. Theintraoral scanner system of claim 3, the computing device further to:generate a virtual model of the three-dimensional intraoral object basedon the registration of the first intraoral image and the secondintraoral image to the model.
 5. The intraoral scanning system of claim3, wherein to register the first intraoral image of the first pluralityof intraoral images relative to the second intraoral image of the firstplurality of intraoral images using the model of the three-dimensionalintraoral object that existed prior to the intraoral scanning, thecomputing device to: determine first rotations and translations based onthe registering the first intraoral image to a first portion of themodel; and determine second rotations and translations based on theregistering the second intraoral image to a second portion of the model.6. The intraoral scanning system of claim 5, the computing devicefurther to: apply the first rotations and translations associated withthe first intraoral image to a third intraoral image of a secondplurality of intraoral images of the three-dimensional intraoral object;and apply the second rotations and translations associated with thesecond intraoral image to a fourth intraoral image of the secondplurality of intraoral images.
 7. The intraoral scanning system of claim6, the computing device further to: stitch the second plurality ofintraoral images with the first plurality of intraoral images based onthe application of the first and second rotations and translations tothe second plurality of intraoral images.
 8. The intraoral scanningsystem of claim 7, wherein a virtual model comprises image data from thefirst plurality of intraoral images and additional image data from thesecond plurality of intraoral images.
 9. The intraoral scanning systemof claim 6, wherein the first plurality of intraoral images compriseoptical images of the three-dimensional intraoral object, and whereinthe second plurality of intraoral images comprise ultrasound images ofthe three-dimensional intraoral object.
 10. A non-transitorycomputer-readable medium comprising instructions that, responsive toexecution by a processing device, cause the processing device to performoperations comprising: receiving a first plurality of intraoral imagesof a three-dimensional intraoral object of a patient generated by anintraoral scanner corresponding to an intraoral scanning of thethree-dimensional intraoral object of the patient; and registering afirst intraoral image of the first plurality of intraoral imagesrelative to a second intraoral image of the first plurality of intraoralimages using a model of the three-dimensional intraoral object thatexisted prior to the intraoral scanning.
 11. The non-transitorycomputer-readable medium of claim 10, wherein the model is based onintraoral data representative of the three-dimensional intraoral objectof the patient.
 12. The non-transitory computer-readable medium of claim10, the operations further comprising: identifying the model of thethree-dimensional intraoral object stored at a data store.
 13. Thenon-transitory computer-readable medium of claim 12, the operationsfurther comprising: generating a virtual model of the three-dimensionalintraoral object based on the registration of the first intraoral imageand the second intraoral image to the model.
 14. The non-transitorycomputer-readable medium of claim 12, wherein registering a firstintraoral image of the first plurality of intraoral images relative to asecond intraoral image of the first plurality of intraoral images usinga model of the three-dimensional intraoral object that existed prior tothe intraoral scanning, comprises: determining first rotations andtranslations based on the registering the first intraoral image to afirst portion of the model; and determining second rotations andtranslations based on the registering the second intraoral image to asecond portion of the model.
 15. The non-transitory computer-readablemedium of claim 14, the operations further comprising: applying thefirst rotations and translations associated with the first intraoralimage to a third intraoral image of a second plurality of intraoralimages of the three-dimensional intraoral object; and applying thesecond rotations and translations associated with the second intraoralimage to a fourth intraoral image of the second plurality of intraoralimages.
 16. The non-transitory computer-readable medium of claim 15, theoperations further comprising: stitching the second plurality ofintraoral images with the first plurality of intraoral images based onthe application of the first and second rotations and translations tothe second plurality of intraoral images.
 17. The non-transitorycomputer-readable medium of claim 16, wherein a virtual model comprisesimage data from the first plurality of intraoral images and additionalimage data from the second plurality of intraoral images.
 18. Thenon-transitory computer-readable medium of claim 15, wherein the firstplurality of intraoral images comprise optical images of thethree-dimensional intraoral object, and wherein the second plurality ofintraoral images comprise ultrasound images of the three-dimensionalintraoral object.
 19. A method, comprising: receiving a first pluralityof intraoral images of a first portion of a three-dimensional intraoralobject of a patient generated by an intraoral scanner corresponding toan intraoral scanning of the three-dimensional intraoral object of thepatient; and registering a first intraoral image of the first pluralityof intraoral images relative to a second intraoral image of the firstplurality of intraoral images using a model of the three-dimensionalintraoral object that existed prior to the intraoral scanning.
 20. Themethod of claim 19, wherein the model is based on intraoral datarepresentative of the three-dimensional intraoral object of the patient.